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Abstract. 

We collect, survey and develop methods of (one-dimensional) stochastic 
approximation in a framework that seems suitable to handle fairly broad 
generalizations of Polya urns. 

To show the applicability of the results we determine the limiting 
fraction of balls in an urn with balls of two colors. We consider two 
models generalizing the Polya urn, in the first one ball is drawn and 
replaced with balls of (possibly) both colors according to which color 
was drawn. In the second, two balls are drawn simultaneously and 
replaced along with balls of (possibly) both colors according to what 
combination of colors were drawn. 
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GENERALIZED POLYA URNS VIA STOCHASTIC APPROXIMATION 1 
1. Introduction 

1.1. Urns. The urn is a common tool in probability theory and statis- 
tics and no student thereof can avoid it. Imagine an urn with w white and b 
black balls. At a beginners level, urns provide examples of how to calculate 
probabilities, e.g. the probability of drawing a white ball is the number of 
white balls divided by the total number of balls, i.e. w/{w + b). If we sample 
more than one ball, say n balls, from the urn and count the number of white 
ball we get examples of the binomial distribution (with parameters n and 
w/{w + b)) and hyper geometric distributions (with parameters w + b,n and 
w/{w + b), depending on whether we sample with or without replacement. 
These distributions in turn are very important in statistical theory as they 
are the key to understanding properties of surveys, e.g. voter polls, such as 
margins of error. 

More aspects of probability theory can be illustrated via urns. Sup- 
pose we draw two balls without replacement. The question "what is the 
probability that the second ball is white?" may introduce the concept of 
conditional probabilities, as the answer depends on the knowledge we have 
(or lack) regarding the outcome of the first draw. Urns are so useful that 
it is hard to imagine an introductory text on probability and statistics not 
ever mentioning urns of any kind. Any reader with a general interest in urns 
may consult |JK77j . 

In 1923 Eggenberger and Polya introduced a new urn model in |EP23j . 
now commonly referred to as a Polya urn. An urn has one white and one 
black ball. We sample one ball and replace it along with one additional 
ball of the same color, and repeat this procedure. It was thought of as a 
simple model for a contagious disease. The first draw might correspond to a 
doctor examining the very first patient of the day. She then has a 50% risk 
of being infected. Now, the essence of a contagious disease is that the more 
people have it, the more likely you are to get it, and vice versa. This is now 
reflected in the model in the following way. Say white ball means "infected" . 
After we draw a white ball we replace at along with one additional white 
ball. Hence, the probability of drawing a white ball next time has risen to 
2/3 ~ 67%. It basically means that the more infected patients the doctor 
gets, the more likely it is that there are yet more to come. Of course, the 
actual numbers in this example is by no means meant to be "realistic" , it is 
rather a qualitative model. 

We can, however, play with the parameters of the model to better fit 
some specific situation if needed. First, the initial composition of the urn 
need not be 1 of each color. A rare disease might correspond to 10 000 black 
balls and only 1 white. Also, some diseases are more contagious than others. 
We could incorporate this by stating that we should not add one additional 
ball, but several, of the same color as the one drawn, corresponding to a 
faster spread of the disease. 
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Any reader interested in Polya urns and generalizations thereof can start 
with |Mah08] . 

Our own interest in Polya-hke urn models comes from a similar situation 
as described above but rather than modelling infectious diseases, it can 
model how something is learned, e.g. a "brain" trying to learn what to do in 
a specific situation. Assume for simplicity that there are only two possible 
ways to act, act 1 and act 2, and that act 1 is the correct way to handle 
the situation and, as such, leads to a reward of some kind. Act 2 is wrong 
and has no benefit for our brain. However, at first it is not known to our 
brain which act is correct (if any). It must somehow learn this by trial and 
error. A very simple urn model describing how this brain could work is the 
following. To model an initial state of ignorance, there is one white ball 
(meaning "do act 1") and one black ball (meaning "do act 2") so that the 
first time it just picks one ball (act) randomly. Then, to model reinforcement 
learning, there is a rule that if an act is deemed successful, more balls of the 
color corresponding to the act just performed are added to the urn. In this 
case; if a white ball is drawn, add, say, one additional white ball and if a 
black ball is drawn replace it but add no more balls. Now, every time our 
brain performs the right act it becomes increasingly likely that it will do so 
again. 

As with the previous model, the interest is mainly qualitative. One 
should not expect that any brain works exactly like an urn. However, it 
captures some of the dynamics of what one can think of as learning; one 
tends to be more likely to do things that have proved successful in the past. 

Again, we can fine tune the parameters. More colors can mean more 
ways to act, different reinforcement rules between colors can specify how 
much benefit the brain gets from the different acts, and so on. 

More specifically, it was questions relating to the so called "signal- 
ing problems" (communicated by Persi Diaconis and Brian Skyrms) that 
spawned the authors interest in these matters. These refer to the situation 
where two (or more) agents try to acquire a common language simultane- 
ously via urns. Recently, one of these problems was solved in [APSV08] 
which also contains a more thorough description of the problem. 

This is some of the motivation behind studying urns evolving along the 
lines of "draw one or several balls and add more balls according to some 
prescribed rule depending on the colors of the drawn balls" . It is also the 
motivation for only looking at the fraction of balls, as these dictate the 
probabilities of "acting correctly" in models of learning. 

1.2. Stochastic approximation algorithms. A stochastic approxi- 
mation algorithm is usually defined as an M'^- valued stochastic process 
adapted to a filtration {^n} such that 



(1.1) 
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holds, where the decreasing "steplengths" 7,1 > satisfy X^nT" ~ ^ 

7n < '^^^ random variables 7„ can be considered stochastic or deter- 
ministic but in either case it is usually assumed that {7ne„} is a martingale 
difference sequence, i.e. 

(1.2) E[7„e„|^„_i] = 0. 

The origin of this subject is jRMSlj . in which Robbins and Monro consid- 
ered the following one-dimensional problem; suppose that given an input x 
to some system in which we get M{x) as output, where M is an unknown 
function and only observable through white noise. What we really observe 
is thus M{x) + e, for some random variable e with Ee = 0. We want to 
find the input 9 so that M{9) = a for some prescribed a. For simplicity we 
might assume that M is nondecreasing and that M[x) = a has a unique 
solution 9. 

A candidate algorithm for finding a sequence {Xn} that converges (in 
some sense) to 6 is to start with some initial input Xq = xq. Given a value 
Xn-, with n > 0, create the next element by 

Xn+l = Xn + ^-r(« - ^i^n) + En+l), 

n + 1 

where —en is the noise associated with the n'th observation. The algorithm 
works on an intuitive level since whenever Xn 7^ 9 then, on average, Xn+i 
takes a step in the direction of 9. 

This describes a stochastic approximation algorithm with drift function 
f{x) = a — M{x) and steplengths 7^ = 1/??,. Of course, there is nothing 
in the formulation of the problem that demands us to set the steplengths 
to 1/n. To demand X^„7n = 00 is natural since this basically means that 
the algorithm can wander arbitrarily far, thus hopefully finding what it is 
looking for, and not converging in a trivial manner. 

Next, since 

n 

Xn - Xq = 7fc(/(Xfc_i) + efc), 
k=l 

the requirement ^„ 7^ < cxo makes VarX„ bounded (under additional as- 
sumptions on the error terms and /). 

In the multidimensional case the heuristics behind the algorithm (II. ip is 
that it constitutes a discrete time version of the ordinary differential equation 

(1.3) 1^* = ^^^*)' 

subject to "noise" . If the noise vanishes for large n it seems plausible that 
the interpolation of X„ should estimate some trajectory of a solution xt 
of p.3p . an idea made precise in |Ben99| . where more references may be 
found. An overview may also be found in |Pem07] . We are however only 
concerned with the one-dimensional case. 

Any reader interested in other aspects of stochastic approximation and 
applications may find [Bor08] useful. 
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1.3. How they fit. Stochastic approximation is very well suited for 
urn models with reinforcements such as the classical Polya urn and gener- 
alizations thereof. If a ball is drawn from an urn and (a bounded number 
of) balls are added according to some reinforcement scheme, the difference 
of the proportion of balls before and after is approximately some function 
of the proportion times 1/n. 

As an example, consider the so called Friedman's urn starting with one 
ball each of two colors where a > balls of the same color and 6 > balls 
of the other color are added along with the ball drawn. The proportion Z„ 
of either color then satisfies 

2 + (n + l)(a + O) '- 

with the drift function f{Zn) = 6(1 — 2Z„) and where "noise" is a martingale 
difference sequence. This resembles the situation considered by Robbins and 
Monro and, as the drift always points towards 1/2, it seems intuitive that 
this is the point of convergence of Z„ (in some sense). That this is so will 
follow from Theorem [1] below. This is "easy" since 1/2 is the unique solution 
of /(x) = 0. 

In other urn models f{x) = may have several roots. There are known 
results that deal with multiple zeros, although often under the property 
()1.2p . Urn schemes where the total number of balls added each time is not 
constant tend to lose this property. We will generalize existing results under 
an assumption slightly weaker than (II. 2p and apply the results to generalized 
Polya urns. 

1.4. A generalized Polya urn considered as a 

stochastic approximation algorithm. First, we will show more precisely 
how stochastic approximation algorithms fit urn schemes by presenting an 
application which will be studied in more detail below. Consider an urn with 
balls of two colors, white and black say. Let Wn and denote the number 
of balls of each color, white and black respectively, after the n'th draw and 
consider the initial values Wq = > and = > to be fixed. After 
each draw we notice the color and replace it along with additional balls 
according to the replacement matrix 

W B 

W / a 6 \ where min{a, 6, c, d} > 
B I c d J ^ and max{a, 6, c, d] > 0, 

so that, e.g. a white ball is replaced along with a additional white and b 
additional black balls. We demand that a, b, c, d are nonnegative numbers. 

This model is by no means new, chapter 3 of [Mah08] gives a historical 
overview. Setting a = d = l, b = c = and Wq = Bq = 1 gives the classical 
Polya urn described in the introduction. 

We let if^^i and l^+i denote the indicators of getting a white and black 
ball in draw n, respectively. We set r„ = Wn + Bn and Zn = Wn/Tn- 
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Recursively, Wn and Tn evolve as 

Wn+i = M/„+oC+i+cI^+i and Tn+i = r„ + (a+6)C+i + (c+d)L^+i 
and hence, with AZ„, = Zn+i — Zn, 

AZn = J—[Wn + aC+1 + cl^+i - Z„(T„ + (a + 6)C+i + (c + 

-tn+l 

^ ■'tW /„ I v A I tB I j\ '7 M _ 



-[C, i(a - (a + + i;? , i(c - (c + 



Let denote the history of the process up to time n, i.e. the cj-algebra 
a{X\, . . . , Xn). We will define 

f{Zn) = E[Yn+l\^n] = ^n(a - (a + + (1 - Z„)(c - (c + 

= aZl + + c, 

where 

a = c + d — a — 6 and (3 = a — 2c — d. 
In the form of a stochastic approximation algorithm we can write 

AZ„ = 7„,+i + Un+l] , 

where = Yn+i - f{Zn) and 7„+i = l/T„+i. 

Now, C/n+i is mean-zero "noise" but \i a + h ^ c + d then in general 
E„7„+i?7„_l_i 7^ 0. However, as will be shown later, |E[7„+iC/n+i|^n]| = 
0{T-'^), and Tn (usually) grows like n, so this conditional expectation is 
vanishing fast. 

2. The method of stochastic approximation 

We will apply the stochastic approximation machinery to fractions and 
thus limit ourselves to processes in [0, 1]. This naturally restricts the noise 
and the function to be bounded. 

2.1. Definition. Stochastic variables are throughout assumed to be 
defined on a probability space (r2,^,P), although we will find no need to 
make any reference to the underlying measurable space ($7, J^). We will also 
consider a filtration {^„,,n > 0} to be given. 

To simplify notation, let E„(-) = E(-|^„,) and P„(-) = 'S'{-\^n) denote 
the conditional expectation and probability, respectively, with respect to 

Definition 1. 

A stochastic approximation algorithm {Xn} is a stochastic process taking 
values in [0, 1], adapted to the filtration {^n}, that satisfies 

(2.1) Xn+l -Xn = 7n+l[/(^„,) + Un+l], 

where 7„, C/„ G / : [0, 1] — )■ M and the following conditions hold a.s. 

(i) ci/n <-fn< Cu/n, 

(ii) \Un\ < Ku, 
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(iii) < Kf, and 

(iv) |E„(7„+iC/„+i)| < Ke7l 



where the constants ci,Cu,Ku,Kf,Ke are positive real numbers. For future 
reference, set K/s. = Cu{Kf + Ky)- 

Remark 1. There is no consensus in the scientific litterature as to exactly 
what constitutes a stochastic approximation algorithm. The main character- 
istic is that a relation of type \2.1\) holds, although the range, measurahility 
etc. of the ingredients 7„, C/„ and f may differ. In this section we state re- 
sults concerning "the" process {Xn} which throughout is understood to he a 
stochastic approximation algorithm according to our definition. 

Remark 2. The condition (iv) could, in view of condition (i), equally well 
have been formulated as |E„(7„_|_i?7„_|_i)| < K'^n~'^, for some positive con- 
stant K'^. The formulation above arises naturally for the applications toward 
the end of this paper. 

Condition (iv) replaces the more common requirement U.^) . so that 
7„,C/„ does not necessarily have conditional expectation 0, but this expecta- 
tion is tending to zero quickly. In what follows, we verify that some results 
known to be true for condition M.2(l carry over to the present situation, as 
well as present some new results. 

2.2. Limit points. In this section we estabhsh that the accumulation 
points of the process {-^n} are a subset of the zeros of /, for continuous /. 
This property is weU known and the ideas for the proofs of Lemma [2] and 
Lemma m are from |Pem07] . Moreover, Theorem [1] gives an existence result 
for the limit of the process 

Let Wn = JkUk so that we may write increments of the process {Xn} 

as 



Lemma 1. {W„} converges almost surely. 

Proof. Set = 'jkUk and = Efc_i(7fc[/fc) and define the martingale 



n-\-k 



Xn+k — X, 



7kf{Xk-l) + Wn+k-Wn. 



k=n+l 



Mn = Eiiyk-Yk). Then 



I 1 ) 1 1 

so that M„ is an L^-martingale and thus convergent. Next, since 





we must also have that X^fc>i Yk converges a.s. 



□ 
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Definition 2. Let 

Xoo = P) {Xm^n+l, ■ ■ ■} 
n>l 

be the set of accumulation points o/{X„}. 

Lemma 2. Suppose that f{x) < —5 (or f{x) > 5), for some 5 > 0, whenever 
X £ (ao, 6o)- Then 

Xoo n (ao,6o) = as- 
and either lim sup„ X„ < ao or lim inf „ Xn > 6o ■ 

Proof. The proof follows that of Lemma 2.6 of |Pem07] . 

Let [a,b] C (ao,6o) and let A = min{a — ao,bo — b} be the smallest 
distance from [a,b] to a point outside (aoi^o)- Let N > 4cui^//A be a 
(random) number large enough so that n,m > N implies 

\Wn-Wm\ < A/4, 

which by Lemma [1] is possible a.s. due to the a.s. convergence of Wn- Then 
we have for any n > N 

Xn+l -Xn = ln+lf{Xn) + Wn+l " W„ < A/2, 

SO that the process after N cannot immediately go from a point to the 
left of ao to a point on the right of a. Also, if n > N, Xn S {a.o,b] and 
Xn+i, ■ ■ ■ , Xn+k-1 G (ao, bo) then 

n+k 

X^+k-Xn= J2 ljf{Xj^l)+Wn+j-Wn 
j=n+l 

n+k 

<-^ 7i + A/4. 

j=n+l 

The last step shows that after N the process cannot increase by more than 
A/4 while inside (ao,6o)) hence cannot escape out to the right. Moreover, 
since "^kyN^k — ^ co a.s., we must have X^+k* < oo for some k* > 0. 

Now, once the process is below ao it will never reach above ao + A/2 
in one step. Once inside (ao,6o) it will never increase by more than A/4. 
Hence, it will never again reach above ao + 3A/4 < a. Obviously, we a.s. 
cannot have both lim inf „ Xn < a and lim sup„ Xn > b. 

The first results follows from choosing [0^,6^] C {aQ,bQ), such that 
Ufc[afc,fefc] = (ao,6o), so that 

P{Xoo n (ao,6o) / 0} < YnXn G [ak,bk] i.o.} = 0. 
The second results follows by an analogous calculation, yielding 
P ( {liminf X„ < ao} n {lim sup X„ > 60} ) = 0, 
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and the observation that since we thus must have hm inf„ > ao or 
hmsup„Xri < 6o) we must in fact have either a.s. 

lim inf Xn > 60 or hm sup X„, < oq , 

n— !>oo n— >oo 

since no accumulation points exist in (ao)^o) by the first result. 

The case where f{x) > 6 on {ao,bo) is analogous. □ 

Next, we introduce the concept of attainability that we need now and 
again to rule out trivialities. 

Definition 3. Call a subset I attainable if for every fixed N >0 there exists 
an n > N such that 

¥{Xn G /) > 0. 

Any "reasonable" stochastic approximation algorithm on [0, 1] should 
have /(O) > and /(I) < 0, otherwise it seems that the drift could push the 
processes out of [0, 1]. The notion of attainability gives a sufficient condition 
to ensure this. 

Lemma 3. Assume that the drift function f is continuous at the boundary 
points and 1. If all neighborhoods of the origin are attainable, then /(O) > 
0. Similarly, if all neighborhoods of 1 are attainable, then /(I) < 0. 

We postpone the proof of this as it will be included in the proof of 
Theorem m on page 1191 

Lemma 4. Suppose f is continuous and let Qf = {x : f{x) = 0} the zeros 
off. Then 

^X^<ZQf} = 1. 

Proof. The continuity of / makes the sets 

An = {xe (0, 1) : fix) > 1/n or f{x) < -1/n} 

open. Hence, each An is a countable union of open intervals, each on which 
/is > 1/n or < —1/n and hence where no accumulation points may exist. 

The only "loose end" here is the boundary. Suppose e.g. that / < close 
to zero (but a priori not at zero). Then it seems that the process might be 
pushed down to zero (or beyond) even though ^ Qf. This is however 
ruled out by Lemma [3l since if neighborhoods of the origin are attainable 
then /(O) > and if they are not, then the process eventually is bounded 
away from the origin. Similarly we can not have / > close to x = 1 and 
attainability of this boundary point simultaneously, unless /(I) = 0. 

It is clear that if / > close to the origin then the process will eventually 
be bounded away from there (and similarly if / < close to x = 1 then the 
process will be bounded away from 1). □ 

Theorem 1. /// is continuous then lim X„ exists a.s. and is in Qf. 

n— !>oo 
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Proof. If lim„X„ does not exist, we can find two different rational 
numbers in the open interval (liminf^ X„, limsup„ X^). 

Let p < q he two arbitrary different rational numbers. If we can show 
that 

P ( {liminf X„ < p} n {limsupX„ > q}] =0, 



the existence of the limit will be established and the claim of the theorem 
will follow from Lemma [H 

To do this we need to distinguish between whether or not p and q are in 
the same connected component of Qf. 

Case 1: p and q are in not in the same connected component of Qf. 
Since Qf is closed and / continuous, there must exist (a, 6) C {p,q) n Q'j 
such that / is non-zero and of the same sign for all x G (a, b). By Lemma [2] 
it is impossible to have lim inf „ X„ < a and lim sup„ Xn > b. 

Case 2: p and q are in the same connected component of Qf. 

Assume that liminf„X„ < p and fix an arbitrary e in such a way that 

< € < q — p. We aim to show that lim sup„ Xn < p + e. 

Recall the notation Wn = YlilkUk- We know from Lemma [1] that Wn 
converges a.s., so for some stochastic > 2K/\/e, we have that n,m> N 
implies \Wn — Wm.\ < e/2. By assumption there is some stochastic n > N 
such that Xn — p < e/2. 

Set 

Ti = mf{k > n : > p} and ai = mf{k > Ti : X^ < p} 
and define, for n > 1, 

Tn+i = inf{k > (7n ■■ Xk > p} and <t„+i = mf{k > t„ : X^ < p}. 
Now, for all k, 

(2.2) Xr, = Xr,-i + AA,,_i <p + KA/Tk <p + e/2. 

Note that f{x) = when x G [p, q]. Hence, if + j — 1 is a time before the 
exit time of the interval [p, q] then 

Tk+j 

Xrk+j = Xr, + 7./(^i-l) + Wr,+j - Wr, = X^, + W^.+j - W^,. 

As 

(2.3) \Wr,+j-Wr,\<e/2 

the process will never be able to reach above p + e before (Jk+i- Since (j2.2p 
and (j2.3p is true for all k, we must have sup;j>„ Xf^ < p + e. □ 
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2.3. Categorizing equilibrium points. Any point x £ Qj = {x : 
f{x) = 0} is called an equilibrium point, or zero, of /. In this paper we 
shall use the following terminology: 

• A point p € Qf is called unstable if there exists a neighborhood rjpp 
of p such that f{x){x — p) > whenever x S 

This means that f{x) > when x is just above p and /(x) < 
when x is just below p, hence the drift is locally pushing the process 
away from p (or not pushing at all). 

If f{x){x — p) > when x £ "^\{p} we call p strictly unstable. 
If / is differentiable then f'{p) > is sufficient to determine that 
p £ Qf is strictly unstable. 

• A point will be called stable if there exists a neighborhood rjpp 
of p such that f{x){x — p) < whenever x S If / is 
differentiable then f'{p) < is sufficient to determine that p £ Qf 
is stable. 

Locally, the drift pushes the process towards p from both di- 
rections. 

• A point p G Q/ n (0, 1) is called a touchpoint if there exists a 
neighborhood ^ of p such that either f{x) > for all x S -y^pXip} 
or f{x) < for all x G ^p\{p}. If / is twice differentiable then 
fip) = f'ip) = ^iid f"{p) 7^ is sufficient to determine that 
p G (0, 1) is a touchpoint. 

A touchpoint may be thought of as having one stable and one 
strictly unstable side. Note that our definition does not allow touch- 
points on the boundary. 

2.4. Nonconvergence. In this section we narrow down the set of limit 
points of the process by excluding certain unstable points. 

2.4.1. Unstable points with non-vanishing error terms. Here we exclude 
the unstable zeros of / as possible limit points, given that the error terms 
do not vanish at these points. For our applications below this is applicable 
to zeros of / in (0, 1) as the noise does vanish at the boundary {0, 1}, a 
problem addressed in the next section. 

Heuristically, the process {Xn} may arrive at an unstable point p G (0, 1) 
by "accident" . To ensure that it does not stay there, we need to know that 
there is enough noise to push the process out into the drift leading away 
from p. 

The main result here. Theorem [2] below, is an adaptation of Theorem 3.5 
of |Pem88] , a sketch of which can be found in |Pem07] and a corresponding 
multidimensional result in |Pem90] . whereby condition (II. 2p is replaced by 
(iv). For results on nonconvergence to more general unstable sets in the 
multidimensional case the reader is referred to section 9 of |Ben99| and 
references there. 

To begin with we mention a result which will be used. 
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Lemma 5. Let A G = <7(U„^„) and suppose there is some integer N 
and a real number < a < 1 such that n > N implies ¥'{A\^n) > a- Then 
= 1. 



Proof. The sequence ¥[A\^n) = IE„(Iyi) is an a.s. convergent martin- 
gale and lim„E„(^) = E{Ia\^oo) = a.s., see Th. 35.6 of |Bil95j . If this 
variable is bounded away from zero it must be 1. □ 

Also, the following will prove to be useful. 

Lemma 6. Let N > be an integer and t be a stopping time with respect 
to the filtration in DefinitionUl such that t > N a.s. Let A G ^oo, B = A^, 

Zk = Zk{N, t) = [Efe_iAXfe„i - AXk-i]I{N<k<T}, and 

m 

Wm= 

fc=Af+l 

Suppose that on A we have Wr > or that we on A have Wr < 0, then 
(2.4) E%[WM]^^^<^nW?<^. 



Proof. First, we note that for any m> N 



oo 



Kl 



EnWI< E^[(AXfc_i)2] < Y ^<^<oo, 

fc=Af+l k=N+l 

SO that Wm is an L^-martingale and hence a.s. convergent. Due to the 
assumption that on A we have WV strictly positive, or strictly negative, we 
must have ¥jy[{A) < 1, otherwise we would have / KnWoo = E]\fWr. In 
particular, this assumption means that ¥i\f{B) > so that we can make the 
following calculation 

= EnWoo = E^iWr] = EN[Wr\A]FNiA) +EN[Wr\B]FN{B) 

Jn{A) 



-EN[Wr\B] = EN[Wr\A] 



El,[Wr\B]=E%[Wr\A] 



^N{Ay^ 



¥NiB)_ 

Next, since EX"^ > E'^X is true for any random variable X, 
En[W^] = En[WMFn{A) + En[W^\B]En{B) 



>Ej,[Wr\A]FNiA)+Ej,[Wr\A] 



En{B) 



E%[WMWn{A){i+^''^^^^ 



Fn{B)J 

2 rr.. , ^,^NiA) 



E%[Wr\A] 



^NiB)- 



□ 
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Theorem 2. Assume that there exist an unstable point p in Qf, i.e. such 
that f{x){x — p) >0 locally, and that 

(2.5) ^nU^+i > Kl 

holds, for some Kl > 0, whenever Xn is close to p. Then 

P{X„ ^ p} = 0. 

Remark 3. The local assumptions f{x){x — p) > and ^2. 5)) can without 
loss of generality be assumed, in the proof, to hold globally. Assume that the 
theorem is proved with global assumptions but that f{x){x—p) > and ^2. 5|) 
are only satisfied when Xn is in a neighborhood jVp of p. Couple the process 
{Xn} after a late time N to another process {Yn}, such that Yat = Xp^ and 

i/{l^,n > N} is constructed so as to satisfy the global assumptions of 
Theorem\^ then so does Yn- Now if¥{Xn — )■ p) > 0, then the same would 
be true of{Yn}, contradicting the theorem. 

Proof of Theorem [2l Following Pemantle's proof there are two steps 
that need verification: 

Step 1: Show that there is a /3 > such that for all N large enough 



(2.6) Fn 



> 1/2. 



sup \Xk-p\ > l3/\fN 

.k>N 

Step 2: Let 

(2.7) r = inf{A: >N:\Xk-p\> P/^/N). 

Conditional on {r < oo} show that 



(2.8) 



inf \Xk-p\ > 13/2-jN 

k>T 



for some a > not depending on N. 
If (ESI) and (ESI) are true then 



> a, 



^Nip^X^) > Fn{t < oo)PJ sup|Xfc-p| > (3/2Vn \{t < oo} ) > ^ > 0, 

V k>T J 2 

and the result follows from Lemma [SI 

Notation: Throughout the proof we will justify inequalities (as they 
appear in calculations) by stating that they hold if a parameter is sufficiently 

large. We will denotes this by <, > or <, > if the inequality holds if n is 
sufficiently large, or if it is clear from the context which parameter is referred 

to, respectively. E.g. ^ + < since this is true if n > 100. 
Verification of Step 1: 

First, in view of Remark [3l we assume that f{x){x — p) > and E^f/^,,.^ > 
Kl holds globally. 
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We aim to show that IPAr{T = 00} < 1/2 where r is defined in (12 .jp . 
Recall that Ka = Cu{Kf + Ku), so that we have the bounds 

|AX„| < and (AX„)2 < ^ 



n + 1 ^ ' - [n + iy 

We may assume that t > N, otherwise there is nothing to prove. Exam- 
ine the process \X^;^m ~ for m > N. An upper bound on this quantity 
is given by 

I^TAm — P\ = \XrAm-l — P + ^XrAm-l\ 

< ^= H — < -7= H < 2 



and so 

4/32 

(2.9) Gn{m) = Ejv[(X.Am - vf] < -jf- 
Next, we make use of the relation 

{XrAm — = [^TAim-l) — P + ^Xm-l^T>mf 

= {^TA(m-l) - Pf + 2(XrA{m-l) " P)^Xm-l^T>m 
+ (AX„_i)^I 

T>m- 

Since m > N we have C =^m-i so any conditional expectation Ejv(-) 
can be calculated as EArEm-i(-)- Hence, 

Gjv(m) = GN{m - 1) + 2E7v{lr>m,(^m,-l - p)E™_i [AXm.^i] } 

(2.10) + E;v{lr>rnE™„l[(AX„_i)2]}. 

Now, by the assumption KnU^_^_i > Kl we get 
E„_i[(AX„_i)2] = E™_i[t^(/(X„_i) + ?7;„)2] 

> ^E^_i[7^/2(X„_i) + 7„f/4 + 2f{Xm -l)jmUm] 



m 
> — 



^/2(X„„i) + ^Em^iUl - 2|/(X„„i)| • |E^„i7^t/„ 
771 m 



> 



771 

cfKL cici2KfK, 



*m 

(2.11) > 



m? m{m — 1)^ 



2m? 

Also, by the assumption f{x){x — p) > we have that 
(X^_i -p)E„_i[AX„_i] 

= {Xm-l - p) f {Xm.^i)Km-l7m + {Xm-l - p)Em-l^mU„ 
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We can now get a lower bound on GAr(m). Continuing ()2.10p . using ()2.1ip 
and combining (j2.12p with the fact that \Xm-i — p\ < P/^fN when A'" < 
m < T, we see that 

GN{m) > GN{m - 1) + J^EN{Ir>.^} - 2 " EN{lr>m} 

'^m \J N{m — ly 

*N c?Kt 

> GN{m-l) + J-^FN{T>m} 



Am? 

(2.13) > GN{m - 1) + -^^n{t = oo}, 



where the last inequality is true for any m since {r > m} D {r = oo}. 
Expanding this recursion gives us 

GjvM >G;v(iV) + |cfKLP^{T = oo} ^ 

k=N+l 

> Gn{N) + ]cfKLFN{T = oo} ^ ^ ^ 



4 ' ' ' \N + 1 m + 1 

Letting m — )■ oo and combining this with (|2.9p we have 

16/32 jV + 1 32/32 

PAr(T = 00) < -g— -— < 



cfKL N - cfKL 



Choosing /3 < ^cfKi/QA makes Fn{t = 00) < 1/2. 
Verification of Step 2: 

Assume throughout that {r < 00}, r defined by (|2.7p . is realized through 
the event {X^- > p + The case when {X^ < p — /3/\/]V} is similar. 

Set 

f = inf{A: > t : Xk < p + P/2Vn}. 

We aim to show that Pt-{'^^ = 00} ^ with a > 0. 

With notation as in Lemma [6] let A = {t < 00} and set = Zk{T,f). 
Notice that by conditioning on r we may consider it fixed (so that Lemma 
5 is indeed applicable). 

Observe that by the assumption f{x){x—p) > we must have f{Xk-i) > 
when t < k < t, since X^-i — p > in this case. This gives us 

c^K 

Efc_iAXfc_i = /(Xfc_i)Efc_i7fc+Efc_i7fcC/fc > " " 
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and hence on the event j4 = {f < oo}, 

f f f 



fe-i 

r+l r+1 T+1 



> 



T+1 ^ ' 



^ clK, /5 ^ ^ ^ 
> -p = +P + 



T-l 2VN VN 

^ /3 clK, *N /3 



2VN N-1 - 4.Vn' 



Lemma [6] now gives us 



f = oo) ^ E^[W^^|f <oo] ^ r/32 /32 



> J9 / > > -r^TPT = a' > 0, 



V(f < 00) - KI/t - NIQK\ - IQK\ 
which imphes Wr{f = 00) > a' /{I + a') = a > 0. □ 

2.4.2. Strictly unstable boundary points. In this section we deal with 
strictly unstable zeros on the boundary. These present a new problem as 
the error terms tend to vanish, making Theorem [2] inapplicable. This new 
result motivated a separate paper |Ren09j . 

Interestingly, the key ingredient here is an upper bound on how fast 
the error terms are vanishing when the process gets near the unstable point 
on the boundary. This is quite the opposite to the situation in Theorem [21 
which required a lower bound on the error terms. This may at first seem odd. 
However, the heuristics is that if the process cannot arrive at the boundary 
in a finite number of steps, knowing that the error terms get small enough 
means an increasing tendency for the process to follow the drift. 

Theorem 3. Suppose of the process {^n} from Definition{l\that G (0, 1) 
for all n. Assume that p G {0, IjnQ/ is such that f{x){x — p) > whenever 
X ^ p is close to p and that there are positive constants K'^K'^ such that 
a.s. 

(2.14) ^nUl+^<K\Xn-p\, 

(2.15) [f{x)f <K'f\x-p\, and 

(2.16) A; • — p| — )• 00, as /c — )• 00. 
Then P{X„ ^ p} = 0. 

Remark 4. Consider the case p = in Theorem\^ In our applications, X„ 
is the fraction of white balls in an urn. If Wn and Tn denote the number of 
white balls and the total number of balls in the urn at time n respectively, 
then Xn = Wn/Tn- What is usually easy to verify is that Tn = 0{n), say 
Tn < Cn, which implies nXn > jjWn so that assumption 112. 1 6]) just means 
that we need that Wn — 00. 
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Proof of Theorem [31 We will, for ease of notation, assume in the 
proof that p = 0. Let e > be a number such that f{x) > if < x < e. 

The idea of the proof is to show that should the process ever be close 
to the origin it is very likely that it doubles its value before it decreases to 
a fraction of its value. So likely in fact, that it will do this time and time 
again until it reaches above e. 

Consider the process {-'^n} after time N. Let A > be a small constant 
and let a £ (0, 1 — A). Define 

n = mf{k >N:Xk> {2Xn) A e} and 
(2.17) fl = inf{A: > N : Xk < uXn}- 

Since we assume that Xn G (0, 1) for all n, we know that Xjsf > and thus 
fl > N. Let T = Ti A f 1 and define the two events A = {fi < ti} and B = 
{ti < fl}. Anticipating an application of Lemma [6l we let Zi^. = Zk{N,T) 
and Wm as in that lemma. 

On the event A we have for any N < k <f\ that X^^i < e and hence 

E,,„iAXfc„i = /(Xfc_i)Efc„i7fe + Efc_i7fcC/fc > ''"^^ 



(A; - 1)2- 

Using this estimate gives us, on the event A, 
fl fl 
Wr = Yl Efc„iAXfc_i - A^fc-i 

N+l N+l 

where the last step is justified by assumption (|2.16p if X^N > '^"^^^ + 1. 
Next, we use assumptions (j2.14p and (I2.15P to get 

(2.18) Efc_i(AXfc)2 < 2Efc_i7|[/2(Xfc_i) + < 

where Ci = 2cl{K'j: + K'J. This in turn gives, since Xk < {2Xn) A e < 2Xn 
whenever k < t, 



^n[W^] < E. 



N 



.A^+l 



2Xn 

k"^ N ' 

N+l 



Since f{x) >OonO<2;<ewe know from Lemma [2] that Xn eventually 
must leave {C,^), for any < ( < e, and hence that B = A'^. So, we can 
apply Lemma M to get 

IP^(^) ^ K[Wr\A] ^ [Xn{1 - a - a)]'N _ [l-X-a]\ ^^ 
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Exploiting that F{A) + F{B) = 1, we see that ([2^9]) is equivalent to, 
with ca = (1 - A - a)V2Ci, 

(2.20) Fn{B) > = 1 > 1 ^ 



1 + CaNXN 1 + CaNXN ~ CaNXN 

Notice that this estimate decreases if a increases. 

Next, define stopping times recursively from (j2.17p 

Tn+i = inf{/c > Tn : Xk > {2Xr^) A e} and 

f„+i = inf{A: > : X^. < aX^} = va.i{k > Tn : X^ < a.n^T„}, 

where a.„ is some (stochastic) number s.t. a^X^-^ = aX]\f and thus an < a 
since either X^r > e (in which case a„ = a) or 

> (2X,„.J A e > {2^Xn) a e > Xjv 

(in which case Tn > N and a„ < a). Define the events = {t^ < f^} and 



stopping times = t^ A ffc. Then ([2:20]) yields, if X^^ > 2^Xj. 



nAk+i\At,^n) > 1 - — ^ > 1 - rw^' 

since Tk > N and < a. If X^-^. > e then F{Af:^i\Ai^, ^Xk) = 1- In either 
case 

F(Ak+i\Ak,^T,) > 1 ; , 

V fe+ll k, Ik) - Ca2^NXN 

holds. 

Now, Hfc^fc is a subset of the event that the process after reaches 
above e. Hence 

p. (sup X, > .) > p„ (n > n (' - ^2^^) 



\k=l / k=l 

El 2 
; — T = 1 > 1, as TV — > OO. 



fc=l 

This contradicts the assumption that P{Xn — ?• 0} > since this requires 
that there is a positive probability that for every prescribed 5 > there is 
an Ns such that n > Ng implies Xn < S. □ 

2.5. Convergence. Now we know when we may exclude some unstable 
points from the set of limit points. Next, we need to check that stability 
of a point p is in fact enough to ensure positive probability of convergence 
to p. After that we also need to know what happens at a touchpoint. A 
touchpoint p' may be thought of as having a "stable side" and an "unstable 
side". Intuitively, one may think that convergence to p' might be possible 
from the stable side, which is indeed the case. 

For the results of the sections to follow we need the notion of attainability, 
recall Definition [3l This is just to rule out trivialities, as there might exists 
a stable point in a neighborhood where the process is somehow forbidden to 
go. Consider e.g. the urn model studied in |HLS80| : an urn has balls of two 
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colors, white and black say, and at each timepoint n there is a proportion 
Xn of white balls and a ball is drawn and replaced along with one additional 
ball of the same color. The probability of drawing a white ball is not Xn but 
h{Xn) where h : [0, 1] — >• [0, 1]. This yields a drift function of f{x) = h{x)—x. 
Consider e.g. h{x) = OifO<2;< 1/2 and define h on (1/2, 1] in such a way 
that h > and a stable zero p of / exists there. Then the attainability of 
neighborhoods of this p depends on the initial condition Xq. If Xq < 1/2 
then p can not be reached as Xn (strictly) decreases to zero. 

2.5.1. Stable points. That convergence to stable points is possible is 
known in related models, e.g. jHLSSO] has a similar result as Theorem 
m below. For related multidimensional results, see section 7.1 of |Ben99j . 

Theorem 4. Suppose p £ Qf is stable, i.e. f{x){x — p) < whenever x ^ p 
is close to p. If every neighborhood of p is attainable then ^{Xn p) > 0. 

Proof. Case 1: p G (0,1). 

We can find a and b such that a < p < b and / > on {a,p) and / < on 
{p, b) . Let 

6 = m.m{b — p,p — a} and e = (5/2. 

Define Aj = {p — e < Xj < p + e}. and let N be large enough so that 
< ±, where C = Kl + IK^cl. 

For k > n, define = (Xfc — p)^. By attainability there exists an n > 
such that P{A„} > 0. Define 

r = inf{A; > n : Xk < a or Xk > b}. 

We want to show that P(t = oo|yl„) is non-zero. 

Notice that on An we have 1^ < and if r < oo then Y-r > 6^. 
On An, for any n < A; < r, we have f{Xk-i){Xk-i — p) < 0, so that 

EnYk = En{Xk-l - p + AXk^if 

= EniXk-l - pf + KniAXk-lf 

+ En[2jk{Xk-i - p)f{Xk-i)] + 2E„(Xfc_i - p)Ek^ijkUk 



fc2 (fc-l)2 

<E„yfc„i + C(A;-l)-2. 
Expanding the above recursion gives a bound on the conditional expectation 

C 



E{Yr\An)<nYn\An)+ ^ 



(A; - 1)2 

k=n+l ^ ' 

(2.21) < E{Yn\An) + < + 



Now, 



n — \ n — 1 



HYrlAn) > EniYrIr<oo\An) > 6^F{t < Oo\A„ 
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and this fact in combination with ()2.2ip yields 

, . N C 111 

P(r < co|A„) < (-) +-p^<- + - = -. 

Hence, P(t = oo\An) > 2/3 so there is a positive probability that 
never leaves {a,b). On the event {r = 00} Lemma [2] implies that limX„ S 
{a,p, 6}. Since we can repeat our argument with any a' £ {a,p) instead of 
a, and any b' G (p, b) instead of b, this implies that limX„ = p (on the event 
{r = 00}). 

Case 2: p£ {0, 1} 

We will prove the statement for p = 0, with p = 1 being analogous. Assume 
that / < on (0, 5*], for some (5* > 0. Set N so large that 4.C/5'^{N - 1) < 
1/12, e* = Aj = {Xj < e*} and r = inf{/c > n : > 6^/2}, 

where n > is such that P(A„) > 0. Analogous to Case 1 we calculate 

En{X^\An) <el + C/{n - 1) and E„(X2|A„) > f P„(t < oo|A„), so that 
P„(r = oo\An) > 2/3. We know that P(A„) > for some n > A^ by 
attainability. 

So, there is a positive probability of the event B = {Xn+k ^ '5*/2 
for all k}. By Lemma [2] it follows that on this event B we must have 
limA„ £ {0,6* /2}. But we may repeat the argument above, choosing any 
6'^ G (0,(5*) in place of 6^, concluding that limA„ € {0,5^/2}. This makes 
it clear that given the event B we must have lim A„ = 0. 

Postponed proof of Lemma [3] 
We will prove Lemma [3] in the case when the origin is attainable, the case 
of the other boundary point, x = 1, is analogous. We assume that / is 
continuous at x = and we need to prove that /(O) > 0. Assume the 
contrary, i.e. that /(O) < and hence, by continuity, that / < on [0, (5*), 
for some 6^ > 0. 

Recall the notation of Lemma [1] and [21 Wn = X]i7fcC^fc- Lemma [1] 
ensures that converges. Hence, for some large (stochastic) N\y > n 

we have that i,j> implies \Wi — Wj\ < 6^/2. 

Prom identical calculations as in Case 2 above, we can conclude that 
there is a positive probability of {Xn+fc < (5*/2 for all k}. Then 

Nw+k 

XNw+k<XNw+ 7j/(Xj„i) + (5*/2 -00, as A; -^00, 

j=Nw+l 

with positive probability, which is a contradiction. Hence, /(O) > 0. An 
analogous argument shows that /(I) < 0, and Lemma [3] follows. □ 

2.5.2. Touchpoints. Theorem [5] below asserts that as long as the slope 
toward a touchpoint p (from the stable side) is not to steep, convergence is 
possible, p need in fact not be a touchpoint as the proof only shows that 
convergence to p may happen from the stable side. In our applications, the 
drift function is differentiable and thus the slope tends to zero, making the 
result applicable. 
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The method of proof is taken from a shnilar result of |Pem9l] . which 
deals with the same urn model as [HLS80] . The interested reader is adviced 
to read this article for more, and stronger, results on touchpoints, albeit not 
in this more general setting of stochastic approximation. 

Theorem 5. Suppose that p is such that K{p — x) < f(x) < for some 
K < 2^ whenever x > p is close to p. 

Also, assume the following technical condition: 

if Suppose there exists some p' > p such that for every N > and 
every y € {p,p') there exists an n > N such that P(X„ > y and 
Xn+i <y)>0. 

[Or similarly suppose that < f{x) < K{p — x) for some K < ^ whenever 
X < p is close to p and assume the existence of a p' < p such that for every 
N > and every y £ ip',p) there exists some n > N such that F{Xn < y 
and X„+i > y) .] 

Remark 5. Condition ^ states that every point in some neighborhood to 
the right - the stable side - of p can potentially he down-crossed at some 
"later" time. 

Proof. First, without loss of generality we make the global assumption 
that f{x) < for x G (0, (remember Remark [3]). The reason that the 

origin is not included in the interval where / is negative is Lemma [31 These 
global assumptions are somewhat superfluous, as we will only be concerned 
with the behavior of the process to the right of p. We will however assume 
that the inequality K{p — x) < f{x) < holds for all x > p. 

The idea here is to show that 

(2.22) F{3N : n > iV implies X„ > p} > 0, 

i.e. that it might happen that the process never again reaches below p. Given 
the event that the process stays above p, Lemma [2] implies that the process 
must converge to p (from above). 

The proof is rather technical and there are numerous constants that 
needs fine tuning in order for everything to work. First, we will use a 
sequence of times < < T„+i oo and a sequence of points 1 > pn > 
Pn+i \ P starting with an index large enough so that pN < p' , where p' 
is defined by condition if. 

We define 

Tat = inf{j > T/v : Xj < p^ < Xj^i} and for n > 

Tn+l = inf{j > Tn : Xj < Pn+l}. 

Notice that by if we have P(TAr < oo) > 0, and if Xi < p for some I > r^r 
then all stopping times are bounded, namely Tn < I, for all n > A^. 

If we can show that P(rn > T„, for ah n > iV) > 0, this wih imply ([222]). 
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For reasons that only become apparent later we set 

(2.23) Tn = exp | ^^^^ ^ j and p„ = p + 

where CuK < Ki < 1/2 and 7 > 1 such that 77^1 < 1/2 and r £ (0, 1) is to 
be specified by the demand that 

(2.24) r„-r2">l, i.e. frexp|^-^l^ > 1. 



127/^1 

If we let g{r) = re'^'^-r)/'2'rKi ^ ^i^^^ ^(^^ ^ ^ ^^^^^ ^/(j) ^ _ < 0, so 

that we know that there exists an r G (0, 1) such that g{r) > 1. From now 

on we fi43 

such an r. Let 

An = {Tn > Tn} and Bn= \ sup Xj <Pn + qn\ , 

[j>Tn J 

where Qn = r'^{'~f — 1) > 0. 
Set 

m 

Zfc = Efc_iAXfc_i - AXk-i and for m > n Wn,m = Z^. 

k=n+l 

We always have the estimate, due to assuming / < 0, 



k (fc- 1)2 - (fc- 1)2' 

where ci = c^K^, and hence on An, for j > Tn-, 

j 



Wr„,j = Yl ]Efc_iAX,,_i-(X,-X,J < + < p^-Xj + -^ 

fc=r„ + l 

We will begin by bounding 

FiB^\An) = P J sup Xj >Pn + qn | A \ 
[j>'rn J 

< P i sup ( p„ + - J- ) > Pn + qn\An 



(2.25) = P \ inf Wr„,j <-qn + TT^\An 



Cl 

" ■r„,J ^ ■li'K I I rp 

Since 1/ [T^J < 2/T„ < 2r2" and qn = r"'{'y — 1) means that we can make n 
large enough to ensure that 

hi{n) = -qn + ci/[Tn\ < 0. 



"'^We may assume that r ^ {1 — jKi Inm/n : m, n £ N} so that Tn ^ N, as it is easier 
to consistently think of Tn as a non-integer. 



22 



HENRIK RENLUND 



Set f = inf{/c > Tn : Wr„,k < hi{n)} and combine the facts that 
nwl^JAn]<Kl/Tn and 

so that we can continue the estimates of (j2.25p 

(2.26) m^JAn) < P(f < oo|A„) < ^r^- 

Notice that on the event Bn, meaning that j > Tn imphes Xj < Pn + Qn, we 
have 

E IE,_iAX,_i= J] E,_i7,-(/(X,_i) + [/,■) 

j=Tn + l j=Tn + l 

>- E j^(^.-i-p)- E iiE.-i7,c/,i 

j=r„ + l i=r„ + l 

> -CuK{pn + qn-p) E 7 " E _ 1)2 - 

Introduce = CuK{pn + Qn — p) = CuK^r"-. By the previous bound, given 
An = {Tn > Tn}, the events A'^_^_^ = {r„+i < T„+i} and S„ together imply, 
first 

^"+1 1 „2 

E > -ciT^r" E --WJ 

j=Tn+l T„+l<j<T„+i+l L nj 

[1] K /<' r2 

> -CuK^r-{lnTn+, - InT^) - ^ - ^ 

[2] 2/r r2 

>-c.r"(l-r)if/Ki-^, 

where [1] is motivated by the fact that if a, 5 G M\N are such that < a 

and b > a + 1, then ^ < ln6 — Ina + 1/a. [2] is motivated by 

a<j<b ^ 

having n large enough since k„ tends to as n grows. Secondly, by setting 

Sn=Pn- < - < K^/Tn, 

Wrn,rn+i= E %-lAX,_i - - X.J 

i='r„ + l 

2/^ r2 

> -C„r"(l - r)K/i^i - - pn+l +Pn-5n 

> ^"(1 _ _ clK/Ki] - C2I \Tn\ = /i2(n). 
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where C2 = 2c^Ke + K^- Notice that 

h2in) = r^C3 - C2/ [TnJ > r^cg - C22r2", 

with C3 = (1 — r)(l — CuK/Ki) > and r < 1, so if n is large enough /^2(^^) 
is positive. 

We have just shown that on the event An we have 

If we let <^ = inf{m > t„, : Wr„,m > h2{n)} then we also have ^^+1 n -B C 
< 00} on An- An upper bound on F{c^ < oo|^n) can be calculated 
analogously to the bound on F{f < oo\An) in ()2.26p . This yields an upper 
bound on P(A^^;^|^„) given by 

mn+Mn) = nA^^, n B'^lAn) + P(A5^+i n BM 

<F(BS,\An)+rkn<00\An) 



TJi\{n) Tnhl{n) 
Kl ( I ^ I 



where 



r„r2'^ \ilin) il{n) J ' 



as ?i — )• 00 since ^^yj. j < r"2r2" — )• 0. Thus we can get the bound 

(J 

for some constant < C < 00. So, 

P(r„ > T„, Vn > iV) = P(r7v < 00) JJ [1 - P(A^+i|A„)] > 0, 

n>N 

since the product converges as 

E F(^^+ll^n) < [,(,)]fi(^(i)2 _ 1) < 



□ 
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3. Generalized Polya urns 

Now, we will apply these result to determine the limiting fraction of balls 
in two related urn models. The stochastic approximation machinery makes 
this fairly easy albeit hard work since the calculations to verify the required 
properties can be rather lengthy. 

3.1. Evolution by one draw. We now return to the model defined in 
Section II. 4[ An urn has Wn white and black balls after the n'th draw. 
Each draw consists of drawing one ball uniformly from the contents of the 
urn, noticing the color and replacing it along with additional balls according 
to the replacement matrix 

W B 

, . W / a 6 \ where min{a, b, c, d} > 

B y c d J ^ and max{a, b, c, d} > 0, 

so that, e.g. a black ball is replaced along with c additional white and d 
additional black balls. The initial values Wq = wq > and Bq = bo > 
are considered fixed, although this makes no difference to the distribution 
of the limiting fraction of white balls, except when a = d and 6 = c = as 
we will see later. 

l^^i and In+i denote the indicators of getting a white and black ball in 
draw n, respectively. We define T„ = Wn + -Bn and Z„ = Wn/Tn and Yn+i 
implicitly by AZ„ = Yn+i/Tn+i, which, after rewriting, gives 

(3.2) Yn+i = {c + d-a- b)ZX.+i + [{a - c)C+i " (c + d)Zn] + c. 

With Yn+i written on this form it is easy to see that the drift function 
f{Zn) = ^nXn+i is given by 

(3.3) f{x) = {c + d-a- b)x^ + (a - 2c - d)x + c. 

By defining Un+i = Yn+i — f{Zn) and 7^ = l/T^ we arrive at the stochastic 
approximation representation 

AZ„ = 7„+i [f{Zn) + Un+l] ■ 

Clearly, / and C/„ are bounded since Z^ G [0, 1], so that conditions (ii) 
and (iii) of Definition [1] are satisfied. 

Condition (i): 

Recall that 7„ = Define 

(3.4) tiiiin = min{a + 6, c + d} and tmax = max{a + 6, c + d}. 
Assume tmin > 0, then 

r„ < To + ntmax =^ n-in > ^ , , > ^ > 

^max -'^ U ^ '^max 

Tn>To + ntmin =^ ?^7n < rp I ^ , — < 7— < oo- 
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Throughout we will assume that t„ 
separately. 



> and handle the case tr. 



Condition (iv): 

To verify condition (iv) of Definition [1] we calculate the expected value of 
Un+i a - {a + b)Zn- f{Zn). 



En 



Tn+1 

Un+1 
_Tn+l 



Tn + a + b 
c+{2c + d 



W , C- {c + d)Zn- fiZn)^B 



+ 



2a 



Tn + c + d 
b)Zn + {a + b-c- d)Zl 



n+1- 



Tn + a + b 



(c - a)Zn + {a + b-c- d)Zl _ 

Tn + C + d ^ 

(a - c)Zn + {2c + d-2a- b)Zl + {a + b-c- d)Zl 

a-b)Zl 



+ 



(c - a)Zn + (2a + 6 - 2c - d)Zl + (c + d ■ 



\C\Zn + C2Z\ + CsZ^ 



r„ + c + (i 

c + d — a 



{Tn + a + b){Tn + c + d) 

for coefficients Ci = a — c,C2 = 2c + d — 2a — b and C^ = a + b — c — d. So, 
there is a constant K(. (depending on a, 6, c, d) such that 



n+l 



< 



7^2 ■ 



The error function 

In order to apply Theorems [2] and [3] via verification of condition (|2.5p and 
(j2.14p we need to calculate what we will call the error function 

t2 



(3.5) 8{Zn) 
One sees from ()3.2p and ()3.3p that 

Un+l = Yn+l - f{Zn) = (C+1 " ^n)^(^n), 

where ^(Zn) = a — c+{c + d — a — b)Zn so that 

3.1.1. Limit points. To determine the limits points of the fraction of 
white balls in this urn model we know from Theorem [1] that we need to look 
at zeros of 

(3.6) f{x) = ax^ + /3x + c, 

where a = c + d— a — b and 13 = a — 2c — d. First notice that 

(3.7) /(O) = c > and /(I) = -6 < 

so that, by the continuity and differentiability of /, there must be a point 
X* G [0, 1] such that f{x*) = and f'{x*) < 0, see Fig. 13.1.11 A unique zero 
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must be the convergence point of the process {Zn} and if more than one 
zero exists, we must check which one is stable (if any). 



We will look at the possible difficult zeros Xu, i.e. the ones that are 
unstable and where the error terms are vanishing, in the sense that £{xu) = 
0, recall ([33]) . 

First, £{x) = if and only if a = c and b = d. This is not surprising since 
there will be no error terms when there is no randomness; this is the urn 
scheme where a white and b black balls are added whatever color is drawn. 
The drift function is then /(x) = —(a + b)x + a so that x* = a/{a + b) is a 
unique (stable) zero. 

It follows from (|3.7p that if at most two zeros exis10 and one of these is 
in (0, 1) then that one is stable. Hence, an unstable zero, if it exists, must 
be at the boundary. By symmetry between colors we need only consider 
unstable zeros at the origin. To that end set c = so that f{x) = ax^ + 
with a = d — a — b and P = a — d, has the property /(O) = 0. In order for 
the origin to be unstable we need parameters to make /(x) > when x is 
very small. We need to consider two cases: 

(i) /3 = but a > 0. This can only happen \i a = d and 6 = i.e. 
/(x) = 0. Having / = makes the sequence {Z^} a (bounded) martingale 
and hence a.s. convergent. This is in fact the classical Polya-Eggenberger urn 
model where it is well known that Zn converges a.s. to a random variable that 
has a beta distribution with parameters w^/a and b^/a, see e.g. Theorem 
2.2 of |Fre65j or Theorem 3.2 of [MahOS]. 

(ii) /3 > 0, i.e. a > d and 6 > arbitrary. The origin is an unstable 
zero and not a convergence point. To see why, we need only notice that 
£{x) certainly can be bounded as (constant)-x and the same is true of /(x). 
Considering Remark HJ Wn — )■ oo is clear since a > does imply that white 
balls are reinforced infinitely often. Hence, Theorem [3] is applicable. 



If there are more than two zeros then / is identically zero, since / is a polynomial 
of order at most 2. 








1 



Figure 1. Schematic picture of /(x) when a > 0. 
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Loose ends 

It remains to check what happens if tmin = niinja + b,c + d} = (but 
a + b + c + d > 0). If c + d = then it is clear that Zn — ?• a/{a + b) a.s. which 
is the unique zero of the driftfunction f{x) = — (a + + ax. 
The case a + 6 = is symmetric. 

So we have proved the following. 

Theorem 6. Consider the Pdlya urn scheme with replacement matrix h3. 1]) . 
starting with a positive number of balls of each color. Then, the limit of the 
fraction of balls exists a.s. Furthermore, apart from the case when a = d 
and 6 = c = 0, in which the fraction of white balls tends a.s. to a beta 
distribution, the a.s. limiting random variable has a one point distribution 
at X* . This point x* is a zero of L3. 6\) in [0, 1] and if two such points exists 
it has the additional property that f'{x*) < 0. 

The author does not expect that Theorem [6] is new (although we have 
never seen it written down). In |Gou89j one finds a similar proposition 
with less generality as it is demanded that a + b = c + d although it has the 
benefit that a and d could be negative. 

It seems likely that Theorem [3] could be proved using only the embedding 
method of Athreya and Karlin into multi-type branching processes, see e.g. 
chapter V of |AK68j . but we have not attempted it. However, the model 
in the next section does not fit this embedding method. 

In (HMPS03] a very general extension of the Polya urn is studied. The 
urn may have balls of several colors and balls are drawn with a probability 
according to a function h of the urn content. At any stage a replacement 
policy is randomly selected from a number of different policies, which may 
include nonbounded random variables, depending on the colors drawn. How- 
ever, their convergence result (Theorem 2.1) is inapplicable to several cases 
in our study due to their assumption of a unique zero of the resulting drift 
function. 

Any reader interested in other types of limit theorems for this model is 
advised to consult |AK68] . |Jan04) and |Jan06| . 



3.2. Evolution by two draws. Again, we will consider an urn with 
balls of two colors but now we turn our attention to an urn scheme where 
two balls are drawn simultaneously and reinforcement is done according to 
which of the three possible combinations of colors this results in. Wn and 
Bn keep their meaning from the previous section but we now assume that 
Wo, bo > 2 so that all 3 combinations of draws have positive probability from 
the start. 
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The replacement matrix becomes 



(3.8) 





W 


B 


WW 




b 


WB 


il 


d 


BB 




f 



where min{a, 6, ...,/}> 
and max{a, b, . . . ./} > 0. 



Prom this we see, e.g. that if we draw a white and a black ball these will be 
replaced along with additional c white and d black balls. 

This model has been studied e.g. in Chapter 10 of |Mah08] . where 
a central limit theorem for the number of white balls (under parameter 
constraints, see remark after Theorem [7]) is presented as well as applications 
of the model. 

Let and I^+i be the indicators of the events that draw n 

results in two white, one black and one white or two black balls, respectively. 
Since balls are drawn simultaneously we have 

^ .WW _ WnjWn - 1) _ „2 ^n(l " ^n) 



Tn{Tn - 1) " Tn-1 



(3.9) E„C+"i = ^fgz^ = - + 2 



j, jBB _ BnjBn — 1) _ /i _ 7 \2 _ -^w(l " ^1 

- ^^^^^ _ 1) - r„ - 1 



Remark 6. // two balls were drawn with replacement we would have the 
simpler situation 

EnC^i = Zl EnC+^=2Zn{l-Zn) and E„/^^i = (1 - Z,)^. 

As Tn ^ oo the rightmost parts of \3. 9fl suggest that for large n there is little 
difference in sampling the two balls with or without replacement. Sampling 
without replacement will make the calculations messier but with the added 
benefit that is it easy to see that the result, Theorem below, will remain 
valid in the simpler case. In the calculations, terms named "Ri" or "Ri{k)" 
are terms that would be zero if we drew with replacement. 

The number of white balls Wn and the total number of balls T„ evolve 
recursively as 

Wn+i = Wn + aC+^ + cCfi + elll, and 

r„+i = T„ + (a + 6)C+^ + (c + ci)lj™i + (e + f)!^!- 

Hence, the increments of the fraction of white balls Zn can be calculated as 

AZ„ = -^([a-(a + 6)Z„]C+^ + [c-(c + d)Z„]lJ™i + [e-(e + /)Z„]I^^^^ 

J 71+1 

which we again denote as AZ^ = Yn-\-\/Tn+i. From the above and (j3.9p we 
calculate 

E„y„+i = g{Zn) + Rn, 
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where 



g{Zn) = aZ.^ + + jZ^ + e and 
Rn = - ^"i^lf"^ (o - 2c + e + aZ,) 

-in J- 



with 



a = -a - b + 2c + 2d - e - f, 

(3.10) /3 = a-4c-2(i + 3e + 2/, and 
7 = 2c - 3e - /. 

Setting Un+i = Yn+i— g{Zn) gives us the stochastic approximation represen- 
tation AZn = jA^[g{Zn) + Un+i]- It is clear that (ii) and (iii) of Definition 
[J are satisfied. 

Condition (i): Define 

(3.11) tinin = min{a + 6, c + d, e + /} and t^ax = {a + 6, c + d, e + /}. 
Assume tmin > 0, then 

r„ < To + nt 

max 

~i~ ^max 

Tn>TQ + nt,-nin =^ njn < 7^ < OO. 

^min 

Throughout we will assume that ti„in > and handle the case tniin = 
separately. 

Condition (iv): 

We write the expectation of Un+i/Tn+i as 



(3.12) E„ 



Yn+l - g{Zn) 



tn+1 



Pl{Zn) ^ P2{Zn) ^ P3{Zn) 



Tn + a + b Tn + c + d Tn + e + f' 



where each pj in (j3.12p has the form 

5 

k=0 

with coefficients cj^^ given by Table [TJ As an example, pi is calculated from 
E„[a — (a + b)Zn — g{Zn)]tl^^. Each Rj{n) is a polynomial in Zn divided 
by T„ — 1, more precisely 

Riin) = ^ — ^[(e - a) + (7 + a + 6)Z„ + /3Z2 + aZ'^], 



Zn{l 


-Zn) 


Tn 


- 1 


Zn{l 


-Zn) 


Tn 


- 1 


Zn{l 


-Zn) 



Mn) = "'^ _/ ^ 2[(c - e) - (7 + c + d)Zn - ^Z^ 



2 aZ^ 



n\ ) 



^3(n) = Z T' [(7 + e + f)Zn + + aZ^]. 



Tn-\ 
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k 




















1 





2c- 2e 


-7 - e - / 


2 


a — e 


-27 - 4c - 2d + 2e 


27 + 2e + 2/ - /5 


3 


—7 — a — b 


27 + 2c + 2d - 2/3 


2/?-a-7-e-/ 


4 


-/3 


2/3- 2a 


2a -/3 


5 


—a 


2a 


—a 



Table 1. Coefficients of the polynomials pj, j = 1, 2, 3. 



We want to show that |E„C/„4.i/r„+i| = 0{X^'^). The Rj terms clearly 
satisfy \Rj{n)/Tn+i\ = 0{T-^). 

Recalling p.lOp . and plugging these in, shows that for each A; = 0, 1, 

0. This gives us 



we have ^ + cf ^ + cf^ 



fc=0 
5 

E 

fc=0 

5 

E 

A;=0 



(1) 



+ 



(2) 



+ 



(3) 



Tn + a + b Tn + c + d Tn + e + f 



(r„ + a + 6)(T„ + c + d)(T„ + e + /) 



.(2) 



.(3) 



(fc)^ (fc) 



+ Co ■'x? + c\'^Tn + c 



and c 



(fc) 



where^(n) = "'"^"^ + + emu 01 

^ ^ T„+a+6 Tn+c+d T„+e+/ i ' '5 

whose exact value is of no importance. This makes it clear that 



,5 



are some constants 



\EnUn+l/Tn+l\ = 0{T- 



The error function 

In order to apply Theorems [2] and [3] via verification of condition (I2.5P and 
(j2.14p we need to calculate the second moment of 

Un+l = Yn+i - g{Zn) = [a - {a + b)Zn] - Zl] 

+ [C - (C + d)Zn]K^i - 2Zn{l - Zn)\ 

Excruciating calculations show that the error function is given by 

e{Z^) = E„C/2+i = Z„(l - Zn)-^{Zn) + Rn, 
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where i?„ is a polynomial in Zn divided by T„ — 1 (so this term tends to 
zero for large n) and ^(x) is a polynomial of order 4 given by 

■^{x) = (a + b-2c-2d + e + ff ■x'^+ 

[-2{a + b-2c-2d + e + f){a-2c + e) 

+ (e + / - a - 6)2 - 4(e + / - c - df] ■ 
[{a -2c + ef + 2{a- e){e + f - a- b) 

-8{c-e){e + f-c-d) + 2{e + f-c- d)'^] ■ 
[{a - ef - 4(c - e)2 + 4(c - e)(e + / - c - d)] ■ x+ 
2{c-ef, 

which is too complicated a formula to work with. Working through the 
expression one can arrive at the form 

(3.13) ^(x) = 2x^{A^ + C^f + x{l - x)bI + 2(1 - xfcl, 
where 

Ax = {-a -b + 2c + 2d-e - f)x + a - 2c + e, 
Bx = {e + f — a — b)x + a — e and 
Cx = (e + / — c — d)x + c — e, 
and the relation 

(3.14) Ax = Bx-2Cx 
holds. 

3.2.1. Limit points. To determine the limiting fraction of white balls we 
need to examine the zeros of 

(3.15) g{x) = ax^ + I3x^ + ^x + e, 

where a = -a-b+2c+2d-e- f , /3 = a-4c-2d+3e+2/ and 7 = 2c-3e-/. 
We see that 

(3.16) g{0) = e > and g{l) = -b<0. 

By continuity and differentiability there will thus exists a point x* with 
f{x*) = and f'{x*) < 0. A difference with the previous model, where the 
urn evolved by a single draw each time, is that we now have more types of 
equilibrium points. Previously, we only encountered unstable zeros on the 
boundary, which we resolved with Theorem [3l Now we will also make use 
of Theorems [2] and [5j 

Remark 7. There will be urn schemes with a unique zero, e.g. the case 
{a,b,...,f) = (3,2,2,3,1,4), where 

g{x) = — 3x + 1, 

and 1/3 is unique. Another example is given by {a,b, . . . , f) = (9, 1, 2, 3, 1, 7) 
where g{x) = — 8(x — 1/2)^. 
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Remark 8. On attainability and condition of Theoreni{5\ 

Consider the replacement matrix [37S[ Let Wn^rhn and bn denote the number 
of times that draws up to time n has resulted in the combinations WW, WB 
and BB, respectively. 
Then 

^ Wq + aWn + crhn + ebn 

To + {a + b)wn + {c + d)mn + {e + f)bn)' 

Since 

P(u;„, = i, rhn. = j, bn = k) > 

for any combination < i,j, k such that i + j + k = n, it follows that any 
open set in [L, U] is attainable, where 

L = mmi — ^ I and U = max \ \ . 
[a + b c + d e + jj [a + b c + d e + j] 

Hence, if Ps is a stable zero of f and p G [-^^, f^] then the conditions of 
Theorem^ is fullfilled and P(Z„ — )■ ps) > 0. 

We can also see that condition of Theorem is satisfied if pt is a 
touchpoint and pt G {L,U). Furthermore, since the drift is continuously 
differentiable, the slope will tend to zero close to pt, making Theorem 
applicable. 

We will not attempt to prove that it will always be the case that neigh- 
borhoods of stable points are attainable and that condition is satisfied 
whenever there is a touchpoint. Our attempts to do so yields too messy 
calculations, but it seems reasonable that this is true. 

In any specific situation there is no problem in verifying these conditions. 

Remark 9. There will be urn schemes where the set of stable zeros contains 
exactly two points and with unstable zeros in (0,1). For example the case 
{a,b,...,f) = (15, 3, 4, 1, 3, 21), where 

g{x) = -2,2x^ + 48x2 - 22x + 3 = -32(x - l/4)(x - l/2)(x - 3/4), 

and 1/2 is unstable whereas 1/4 and 3/4 are stable. Notice that L = 3/24 < 
1/4 and 3/4 < [/ = 15/18 so that both stable points are possible convergence 
points by Remark\^ 

Remark 10. Touchpoints may arise. If {a, . . . , f) = (35, 9, 1, 1, 3, 21) then 

g{x) = -64x^ + 80x2 _ 28x + 3 = -64(x - l/^f{x - 3/4), 

where 1/4 is a touchpoint and 3/4 is stable. Notice that L = 3/24 < 1/4 < 
3/4 < 35/44 = [/ so that both the stable point and the touchpoint are possible 
convergence points by Remark\^ 

No unstable equilibrium in (0, 1) with vanishing error terms. 

Now we will examine whether there could exist an unstable zero x„ in (0, 1) 
such that £{xu) = 0, i.e. an unstable zero to which we can not apply Theorem 
El We recaU that £:(x) = x(l - x)*(x) + i?„ where i?„ = ©(T^^) and ^ 
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is given in (I3.13p . Hence, we need to look at points x G (0,1) such that 
^(x) = 0. 

First, ^l(x) = Q \i Ax = Bx = Cx = ^- It is easy to calculate and 
intuitive that this can only happen \i a = c = d and b = d = f , since this is 
the case when there is no randomness involved in the urn scheme, a white 
and b black balls are added whatever is drawn. Then g{x) = — (a + b)x + a 
so that X* = a/{a + b) is unique. 

Next, we need to solve ^{x) = for < a; < 1. We will do this by going 
through the cases when exactly one of Cx,Bx or Ax, or none, is zero. This 
suffices due to relation (|3.14p . 

Note, since the drift function is a polynomial of order at most 3 with 
boundary condition (j3.16p . the only time when an unstable x* G (0, 1) has 
g'{x*) = is when g{x) = 0. This case is special and will be treated below. 
Hence, we need only verify that if x* € (0, 1) is a point where £■ vanishes, 
then X* is not strictly unstable, i.e. that g'{x*) < 0. 

The case Cx = 

If Cx = 0, i.e. c = e and f = d, then from (|3.14p we have that Ax = Bx = 
[e + f — a — b)x + a — e. Then 



^'(x) = 2x^AI + x(l - x)Al = x{l + x)Al. 



We assume Bx is not identically zero, soife + / = a + 6 then Bx = o — e is 
never zero. Ife + // a + 6 then Bx* = for x* = ^.^fZl-b = "^^^ ^^^i^* 
function g{x) is now 

g{x) = {-a ~b + e + /)x^ + (a - e)x^ - (e + f)x + e 
= Vx^ - Mx^ - (e + f)x + e, 

so that g{x*) = e - and thus g{x*) = ii af = eb. If e / then 

e + f - a- af/e 
= (l-a/e)^-(e + /)<0. 

If e = 0, then a/ = so that x* = or x* = a/(a + 6). In the latter case we 
have <7'(^) = -aV(a + 6) <0. 

The case Bx = 

If Bx = then a = e and f = b, Ax = —2Cx and 

^'(x) = 2x'^{-Cx)^ + 2(1 - xfC^ = C72(4x2 - 4x + 2), 

where 4x^ — 4x + 2 has no roots in (0, 1). We assume that Ax = —2Cx is 
not identically zero so if a + 6 = c + ci then Ax = 2(a — c) is never zero. If 
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a + bj^c + d then A^.. = for x* = = ^. Now 

g{x) = 2{-a -b + c + d)x^ + 2(2a + 6 - 2c - d)x'^ + (2c - ?>a - b)x + a 
= -2Vx^ + {2N + 2V)x^ - {2N + a + b)x + a, 

so that g{x*) = a — = if c6 = ad. If a 7^ then 

g'(x*) = -2 [° ~''^\/ +a-2c-b 



a + b — c — cb/a 



a 



2 



a^ + b'^ + 2bc 



= -2(1 - c/a) + a - 6 - 2c = < 0. 

a + b a + b 

If a = then c6 = so x* = or x* = c/ {c + d). In the latter case we have 
g'{x*) = -2cd/{c + d) < 0. 

The case = ^ 

If = then 2c = a + e, 2d = b + f , Bj, = 20^ and 

^(x) = 2x'^Cl + x(l - x)(2C^.)^ + 2(1 - xfCl = 2Cl 



We assume that Bx = 2Cx is not identically zero so if e + / — a — 6 = then 
Bx = 
and 



Bx = a — e is never zero. Ife + // a + 6 then Bx* = for x* = g_)_j_°_j, ^ 



so that 



g{x) = {e + f — a — b)x^ + (a — 2e — f)x + e 
= Vx^ - [N + c + f)x + e, 

g'{x*)=Af-e- f = -a- f <0. 



The case Ax,Bx, Cx not = 

Suppose a e ^ c, a+b / e+/ 7^ c+d, 2c / a+c and a+b+c+d 7^ 2c+2d. 
The only chance of having ^(x) = 0, for x € (0, 1), is for Ax, Bx and Cx to 
be zero simultaneously. A common zero x* of Ax, Bx and Cx when none of 
these is identically zero imposes 

^ e — a e — c a — 2c + e 

^ ~e + f-a-b~e + f-c-d~a + b-2c-2d + e + f 

which is the case whenever e{b — d) + c{f — b) + a[d — /) = 0. 

Setting X* = {e - a) / {e + f - a -b) and d = [b{e - c) + /(c - a)]/(e - a) 
and using Maple yields the simple expression 

f af-eb 

e + J — a — b 

which is zero if af = eb. The derivative simplifies to 
g{x ) = -a — - (/ + 2c)- 



a + f-b 'f-b + e 
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If o = then 6e = so g'{x*) is -(/ + 2c) or -(/ + 2c)j^. If a / we can 
write f = be/a and 

a + a + 

So we can determine that there is no strictly unstable zero of g{x) in 
(0, 1) such that the error terms are vanishing. 

Unstable boundary points 

Next, we check the boundary. By symmetry of colors we need only con- 
sider an unstable zero of g at the origin. To that end set e = so that 
g{x) = ax^ + + 72; has g{0) = 0. We check the cases where g > close 
to the origin: 

(a) 7 = 0, /3 = and a > is only possible if 2c = f, 2d = a and 6 = so 
that a = and hence g{x) = 0. It is then, in some sense, the 2-draw version 
of the classical Polya urn. The special case oi 2c = 2d = a = f has been 
studied in [CW05j and they show that the limiting variable has an abso- 
lutely continuous distribution. They also include a simulation study that 
indicates that the limiting distribution "resembles" the beta distribution. 

By Theorem [21 we may only conclude that the limiting distribution has 
no point masses on (0, 1). 

(b) 7 = and /3 > 0, i.e. 2c = f and a > 2d. 

(c) 7 > 0, i.e. 2c > /. 

In both (b) and (c) the conditions of Theorem [3] are satisfied. kZf^ can be 
made arbitrarily big since Wk — )• 00. In the second case this is due to c > 0. 
Since the combination WB will always be drawn infinitely often, it follows 
that white balls will tend to infinity. In the first case, either c > and we 
are done, or c = / = e = and a > so that the combination WW will be 
drawn infinitely often. 

Also, both g{x) and £{x) behave like (constant) -x when x is close to 
zero. Thus, convergence to a strictly unstable boundary point is impossible. 

Loose ends 

Here we examine what happens if = min{a -|- 6, c -|- d, e -|- /} = 0. 

1. c = d = e = f = has drift g{x) = —{a + b)x^ + ax^. It is clear that 
Zn does converge to a/ {a + b) a.s. (i.e. the stable zero of g) since any draw 
that alters the urns composition does so by increasing the number of white 
balls by a and the numbers of black balls by b. 

2. a = b = c = d = Ois symmetric to the above case c = d = e = f = 0; Zn 
will converge to the stable zero of g. 

3. a = b = e = f = has 

g{x) = 2(c + d)x^ - (4c + 2d)x'^ + 2cx = 2x{l -x)[c-{c + d)x] 

with c/ (c -|- d) being the only stable zero. It is clear that Z„ converges to 
this point a.s. 
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4. a = b = and min{c + d, e + /} > so that nothing happens when 
two white bahs are drawn (except that they are replaced in the urn). Let 
Ti = mf{k > 1 : Tfc > Tfc_i} and for n > 1 

By looking at the sequence we ignore the times when two white balls 
are drawn. This makes no difference to the limit since -^r„+fe = -^t„ for 

< < Tn+l - Tn- 

However, since 

W tBB ^ ^rn(TV„-l)' ^ - 1 , 

^^"■"^n + l Br„{B^„~l) ^ 2Wr„Br„ + 2Wr„ - 1 

^T"n (^Tn — 1) ('^Tn ~l) 

^ jWB _ 



it is more convenient to define r„ = B^^ + 2Wr„ — 1 and Z„ = 2WT-„/Tn. It 
is straightforward to compute AZ„ = y^+i/T^+i where 

Yn+i = Ij^+,(2c - (2c + + I^„^^^(2e - (2e + 

so that 

g{Zn) = Er„Yn+i = (2e + / - 2c - d)Zl + (2c - 4e - + 2e. 
We also get the error function, with f/n+i = i^+i — g{Zn)-, as 

£{Zn) = ^rr.{Ul+i) = [(2e + / - 2c - + 2c - 2e]2z„(l - 

Since we have assumed min{c + d, e + /} > we know that (i) of Definition 
([1]) is satisfied, and one easily verifies (ii)-(iv). 

As 5(0) = 2e > and g{l) = —d < any unstable zero of g must be on 
the boundary {0, 1}. We can apply Theorem [3] to conclude that Zn will not 
converge to an unstable boundary point. 

So, we have the original process Z^ = Wn/ {Wn + Bn) described by the 
drift function 

g{x) = (2c + 2c - e - + (-4c - 2d + 3e + 2f)x^ + (2c - 3e - /)x + e 
= (1 - x)[(-2c -2d + e + f)x^ + (2c - 2e - f)x + e] 

and our "new" process Z„ = Wr„/{2Wr„ + -Bt„ — 1) described by 
g{x) = (-2c -d + 2e + f)x^ + (2c - 4e - f)x + 2e. 

Now, as r„ — )• 00, the limit x = lim„ Z„ and y = lim„ Zn are related as 

2x y 
y = and x 



x+1 2-y 
In particular x = is equivalent to y = and x = 1 is equivalent to y = 1. 
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A straightforward calculations shows that 

1 , n9 ^ / 2x \ q(x) 
(3.17) 7t(1 + ^) ^' ^- ' 



so that g and have the same zeros, with the possible exception of x = 1. 
For X = 1 we examine two cases: 

(i) If d > then ^(1) = d > 0, whereas g(X) = always, so that 
1 ^ Zoo, which implies 1 ^ Zqo- But ^'(l) = d > so that noncon- 
vergence to 1 is what we would expect.. 

(ii) If d = then ^(1) = g{l) = 0. We examine the behavior of these 
close to 1 via the calculations 

^(1 - e) = e[2ee+ (2c- /)(! - e)] and 

<^(l_e) = e2[e, + (2c-/)(l-e)]. 

Hence x = 1 is stable/unstable simultaneously for g and g. 
Differentiating (j3.17p for x ^ 1 yields 

{l + x)g{^^] +g' ) = —^g{x) + -^g'{x), 



^x + lj \x + \J (1 — x)2 1 — X 

so that at any point x < 1 where g and g vanishes we have 

w 2x \ 1 
\x + l/ 1 — X 

i.e. g and g have the same stable and strictly unstable points (if any). For 
any x < 1 such that g' and g' vanishes we also have 

9 — — r = 9 [x), 

\x + 1/ 1 — X 

so that g and g have identical touchpoints (if any). 

Thus the convergence of Zn to a stable zero of g implies the convergence 
of Zn to a stable zero of g. Also, the non-convergence to an unstable zero 
of g implies the non-convergence to an unstable zero of g. 

5. e = / = and min{a + 6, c + d} > is symmetric to 4. 

6. c = d = and min{a + 6, e + /} > 0. Define as in the previous section 
and Zn = Zt-„. Now 



-"+1 WrSWr^-l) + B,„{Br„-l) 

is a difficult expression to work with directly, so we rewrite it as 

E^JBB ^ Zl Rn ^^^^^ 

^2 + (1 _ ZnY Tn 

Z„(l/2 - Zn){l - Zn) 



Zl + {l-ZnY][Zl + {l-ZnY-l/Tn] 
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Next, set Yn+i = Tn+iAZ„. Then 



Yn+i = {e + f-a- h)ZnK^^, + (a - e)I 



rBB 

Tn + l 



(e + f)Zn + e, 



so that 



g{x) = a 



a 



+ (1 — x)2 
e + f — a — h. 



Rn[aZn + (g - e)] 

fn 

3 

+ (a - e) 



where 



+ (1 — x) 



+ (— e — /)x + e, and 



Next, 

Un+l = Yn+1 - g{Zn) = [aZn + (o - e)] 
which yields the error function 

£{Zn) = ErM+i = [aZn + (a - e)]2 



tWW 

Tn + l 



z^ + {i-z^y 



+ 



Rn[l-2Zl/{Zl + {l-Zn?)]-^ 



One may also verify (iv) of Definition ([T|) by calculating: 

Un+i 1 a[aZn + {a- e)] 



■■'Tn+l {Tn + a + h){Tn + e + f) [Z2 + (1 - Z^T 

which certainly is 0{T~'^). 

Next we compare Z„ with the original process Z„,, with drift 

g(x) = {-e- f -a- b)x^ + (a + 3e + 2f)x^ + (-3e - /)x + e. 

It is straightforward to verify that 

(3.18) [x^ + {l-xf]g{x)=g{x), 

so that g and g have the same equilibrium points. Differentiating (j3.18p 
yields 

(4x - l)g{x) + [x^ + (1 - xf]g\x) = g\x), 
so, at any point where g = g = we have sgn[^'(x)] = sgn[g'{x)]. Differenti- 
ating again at a point where g' = g' = yields [x^ + (1 — x)'^]g"{x) = g"{x). 
In conclusion, g and g have "similar" equilibrium points in that they are 
stable, strictly unstable, or touchpoints simultaneously. 
We have proved the following. 

Theorem 7. Suppose that the Pdlya urn scheme of drawing two balls (with 
or without replacement) according to \3. 8\) has wo,bQ > 1 (or just WQ,bQ > 
if drawn with replacement). Then, the limit of the fraction of balls exists a.s. 
Furthermore, apart from the case a = 2d, f = 2c and 6 = e = 0, in which all 
we may conclude is that the limiting variable of the fraction of white balls 
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has no point masses in (0, 1), the limiting random variable has support only 
on the zeros of g, defined in \3.1^) . such that the derivative g' is nonpositive 
there. 

In Theorem 10.1 of |Mah08] one can find a central limit theorem for 
the number of white balls in the urn scheme [3.81 with the added constraints 
of a constant row sum larger than one (i.e. a + b = c + d = e + f = K> 1) 
and a — 2c + e = 0, which together has the effect that the drift function 
becomes linear (i.e. a = (3 = in I3.10p . It is also noted there (Proposition 
10.3) that the fraction of white balls converge (in probability) to — e/7. 
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